Field of the Invention
This invention relates generally to Peripheral Component Interconnect Express (PCIe) and particularly to sharing PCIe end points across servers.
Background
The Peripheral Component Interconnect Express (PCIe) Specification allows only one host (one root) in the PCIe hierarchy. The PCI Multi Root IO Virtualization (MRIOV) Specification addresses how a PCIe end point, an example of which may be a High Bandwidth Input/Output (IO) resource, can be shared across multiple servers/hosts. Today, there is no support available for MRIOV in the entire eco system (eco system consisting of hosts, drivers, switch, and PCIe devices). There has been an attempt by the PCIe switch manufacturers to address this by adding a Non Transparent Bridge (NTB) ports to the PCIe switch. The existing solution however requires management software on all the servers and the “Shared PCIe subsystem” (PCIe subsystem consists of PCIe switch with NT ports, a local processor which acts as the root complex and the PCIe device). The management software is required because the server enumerates the PCIe end point as a Non Transparent Bridge port. The driver does not know what to do with the NTB. The device drivers for the end point behind the switch that exists on the server do not get loaded automatically. The Management Software on the server and on the shared PCIe subsystem have to communicate with each other and share the capabilities of the PCIe subsystem.
Another issue with using a PCIe switch with NT ports is that the entire memory space of the server cannot be exposed to the switch, rather, only a portion of the memory on the server is available for data transfer. In a PCI system, the end point can access the entire memory space of the server, which requires the server to move the data into the memory space mapped for the end point to access. This requires extensive changes to the existing device drivers and adds to the latency.
What is needed is a device and method for enabling sharing of a PCIe end point across multiple servers as a plug-n-play device.